{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Monte Carlo(MC) Estimation\n",
    "\n",
    "We will be using MC estimation on sample 4x4 grid world  \n",
    "\n",
    "![GridWorld](./images/gridworld.png \"Grid World\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Monte Carlo Prediction for Estimation (\"first-visit\")\n",
    "\n",
    "Monte Carlo Prediction is carried out by sampling the trajectories over many episodes and using the rewards seen in samples as estimate for state values. The backup digram is given below. Pseudo code for the algorithm is given in Fig 4-2 in the text.\n",
    "\n",
    "![MC backup](./images/mc_backup.png \"MC Backup\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "10:1: E402 module level import not at top of file\n"
     ]
    }
   ],
   "source": [
    "# Initial imports and enviroment setup\n",
    "import numpy as np\n",
    "import sys\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "sns.set()\n",
    "\n",
    "# create grid world invironment\n",
    "from gridworld import GridworldEnv\n",
    "env = GridworldEnv()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# MC Prediction\n",
    "\n",
    "def mc_policy_eval(policy, env, discount_factor=1.0, episode_count=100):\n",
    "    \"\"\"\n",
    "    Evaluate a policy given an environment.\n",
    "\n",
    "    Args:\n",
    "        policy: [S, A]shaped matrix representing the policy. Random in our case\n",
    "        env: OpenAI env. In model free setup you have no access to env.P, t\n",
    "             ransition dynamics of the environment.\n",
    "             use step(a) to take an action and receive a tuple of (\n",
    "             s', r, done, info)\n",
    "             env.nS is number of states in the environment.\n",
    "             env.nA is number of actions in the environment.\n",
    "        episode_count: Number of episodes:\n",
    "        discount_factor: Gamma discount factor.\n",
    "\n",
    "    Returns:\n",
    "        Vector of length env.nS representing the value function.\n",
    "    \"\"\"\n",
    "    # Start with (all 0) state value array and a visit count of zero\n",
    "    V = np.zeros(env.nS)\n",
    "    N = np.zeros(env.nS)\n",
    "    i = 0\n",
    "\n",
    "    # run multiple episodes\n",
    "    while i < episode_count:\n",
    "\n",
    "        # collect samples for one episode\n",
    "        episode_states = []\n",
    "        episode_returns = []\n",
    "        state = env.reset()\n",
    "        episode_states.append(state)\n",
    "        while True:\n",
    "            action = np.random.choice(env.nA, p=policy[state])\n",
    "            (state, reward, done, _) = env.step(action)\n",
    "            episode_returns.append(reward)\n",
    "            if not done:\n",
    "                episode_states.append(state)\n",
    "            else:\n",
    "                break\n",
    "\n",
    "        # update state values\n",
    "        G = 0\n",
    "        count = len(episode_states)\n",
    "        for t in range(count-1, -1, -1):\n",
    "            s, r = episode_states[t], episode_returns[t]\n",
    "            G = discount_factor * G + r\n",
    "            if s not in episode_states[:t]:\n",
    "                N[s] += 1\n",
    "                V[s] = V[s] + 1/N[s] * (G-V[s])\n",
    "\n",
    "        i = i+1\n",
    "\n",
    "    return np.array(V)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Custom print to show state values inside the grid\n",
    "def grid_print(V, k=None):\n",
    "    ax = sns.heatmap(V.reshape(env.shape),\n",
    "                     annot=True, square=True,\n",
    "                     cbar=False, cmap='Blues',\n",
    "                     xticklabels=False, yticklabels=False)\n",
    "\n",
    "    if k:\n",
    "        ax.set(title=\"K = {0}\".format(k))\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOcAAADnCAYAAADl9EEgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAASt0lEQVR4nO3deVhUdd/H8c+wKAikiCCKEuKSirigKO5iuBVgamJUpLg84YIL3Wpm5oItLrfe5ZIptmi5YOmjWW65Z0qiprjhhrKJqCguhDLDef6w5oFL4I4BzvmO1+f1F/M7x2u+R3nPmRmv6xydoigKiEgcC60HIKKiMU4ioRgnkVCMk0goxkkklFVJG21bjVFrDlVNmTte6xEqhE8tB61HqDAe1ey0HqHCNKtjX+Q6z5xEQjFOIqEYJ5FQjJNIKMZJJBTjJBKKcRIJxTiJhGKcREIxTiKhGCeRUIyTSCjGSSQU4yQSinESCcU4iYRinERCMU4ioRgnkVCMk0goxkkkFOMkEqrES2OqpXcnL8yKDEblSlY4fTENETPX4P7DXK3HMpmiKDi8eiGq1X4eTQMGAAD0jx/haOznuH3tAqAocPJ4Ab4hI2FVqbLG05aOoihYt/gjuLp7wr9vqHF92pAgVHNyNj7u1vc1tO7SU4sRTaIoChbPmQ53zwboG/IWAMBgMCBm0RycPXkcAODTriPeens8dDqdKjNpHmcNR3t8MfNNdA9fgMvJNzF7bF9Ejw3G+I9jtR7NJNkZyTga+zluXU1EtdrPG9dP71gPJd+Al6cshgIFv30zH2d2xqJFYJiG05bOjdSr2LhiIZIvnoOru6dxPTMtGVXsHfDOv7/UcDrTpV5LworPPsHF86fh7tnAuL5/109IT7mGBTHroSj5eG/sUBw+8As6dO2hylyaxxng1xjHzlzD5eSbAIDlGw7i9/VTzDbOCwd+Qv32vWDn6FxovWaDZrCrXhM6CwvoADjWqY/s68naDGmiQ9s2oV1AIKrVqFlo/WriaVhYWGDx+2OQ+/ABmrfvhoABYbCwtNRo0tLZtjkWAS+9ghouroXW8/Pz8Sj3T+jzHiNfUaDPy4O1tXrvdDSPs46rI1Jv3DU+Tsu8i6oOtnCwszHLt7a+ISMBABnnjxdar9XEx/jzg6xMJO7djLahkarOVlb9R0wAACT+cbTQer7BgIbN2+DlN99GvkGPmA8nw6ZKFXQJDNFizFIbMXYyAOCP+COF1v17BeHwgV8wYlAfGAwGtGjjB98OXVSbS/MvhHQ6HYq6f6/BkK/BNBXvdvJF7Fo4CY26BqKOd1utxykXfj2C0H/4eFS2sYWtnQO6BoUgIe6g1mOVWeyq5XiuqiNWfr8Ly9f9jAf3srEldrVqz6/5mTMl4w58vT2Mj91cqiIr+yFych9rN1QpnNy6GmkJcQAAN+92JX6GvBq/H0djl6LNwJGo59tNpQlNt33tSpyJPwQA8GrTEb1DhxW5X/y+Hajt0QC1PeoDABQAlpaa/2oVa+1XnyP+8AEAQJv2XRAaPrLI/eJ+3YthYybC2toa1tbW6NYzEEcO7EZwiDrfE2j+N7j78Dl8EtUP9d2dcTn5Joa/2hlb9yVoPdY/1iIw7B99qZOaEIf4779A99Gz4fR8QxUmK7veocOKDbKgjJQrOHVkP4ZMjIZBr8ehbRvh01mdL01MERo+stggC/Js2Bi/7d8F71a+0OvzEH/4ABo19VZhwic0j/PmnQd4e8a3WDNvGCpZWeFK6i0Mn7ZK67HK3fFNKwEoiFvzqXGthmdTtB00SruhyknPkHBsjPkP5kcNgUGvR4sO/mgXEKj1WGUWPjIKKxbNReSQ/rCwsERzH1/0HTRYtefXKUV94PsLbwFoXngLQPPEWwASmRnGSSQU4yQSinESCcU4iYRinERCMU4ioRgnkVCMk0goxkkkFOMkEopxEgnFOImEYpxEQjFOIqEYJ5FQjJNIKMZJJBTjJBKKcRIJxTiJhGKcREKVeN3akTOfzUtjNnGuovUIFaKWva3WI1QY28rmcVOk8sQzJ5FQjJNIKMZJJBTjJBKKcRIJxTiJhGKcREIxTiKhGCeRUIyTSCjGSSQU4yQSinESCcU4iYRinERCMU4ioRgnkVCMk0goxkkkFOMkEopxEgnFOImEYpxEQpV43Vq1hLaqhev3HmHf5Szjmo2VBcZ0cse6ExlIzc7VcLrSUxQFPyz9BK7unugUNOip7WvmfwCH6k4IGjpOg+nKRlEULJs/E3U96iNwYBgA4D/Rk3EjPcW4T2ZGOpo098G/Zi7QasxSUxQFCz6cBg/Phhjw+mAAwP172Vg8/0NcuZgIG1tb9HipL4JfDVVtJk3jdLGvhAHeNeHuaIvr9x4Z15u42KFvMxdUt7XWcDrTZKZew49fforUS+fg6u751PaDm9fi6vlT8O7gr8F0ZZOWnISvFs/F5fOnUdejvnF9/LQ5xp8vJ57Bp7PfRfjoSVqMaJLkq1ewdMHHSDybAA/Phsb15Z/Ng61tFSz7diPy8/MRPWU8atZyQ7uOXVSZS9O3tZ3qOSIuORsn0+8XWu/s6Yjvjl/HvUcGjSYzXdzO/0Wb7i+hmV/Xp7ZdOfMHLpw8irY9gjWYrOx2btkA/9590a7Li0Vu1+flYdn8mQiLiIKTi6vK05lu68b16BXYD539exRav5R4Dt17vQxLS0tYW1vDt31nHNq3S7W5NI1zY8INHE+799T68iOpSLlrXm9l/xY0dBxadAp4av1e1i38/PUihEROhc7CPD/qh4+ZhI7dexe7fe/2zahWvQZ8O5rXu4JRUVPg3/Olp9ZfaOqNPTt+gl6fhz9zcnBo/25k3b6l2lzm+VtiZgx6PWI/i0afwaPh4Oik9TgVZtumtej3+jCtxyg3w8dEATodIsNfQ/SUCWjl6wcrK/U+aqn6mbP3CzXg5WoPADiT8QDbE9V7Faoov8R+ifPxvwEAGrfpgICQoU/tk3YlEVmZ17Ft1VIAwIO7WcjPz4f+8WP0i5io6rylseGbZTh+5AAAwMevCwYOjih236uXEmEw6NGkuY9a45lsdcxSxP26DwDQrlM3hA0fVeR+OQ8fYtio8XB4rioAYP2qGNSuU1etMdWNc3virWciyIICQoYWGWRB7o28MGlprPHx7g1fI+d+tvhvawcOjigxyILOnToGr5a+0Ol0FTxV2YUNH1VskAX9vHkDch4+xKioKbiTdRs7ftyEd2fN+a9/rryI+K8UMn8Z6SlwrllL6zHKVUjYMMyPnoqRYQOgKAreHD4KjZo0U+35dYqiKMVtjNpyXrVB1NTe/TmtR6gQntXstR6hwlSzM7//Vvun6jsXfV9VfiFEJBTjJBKKcRIJxTiJhGKcREIxTiKhGCeRUIyTSCjGSSQU4yQSinESCcU4iYRinERCMU4ioRgnkVCMk0goxkkkFOMkEopxEgnFOImEYpxEQjFOIqFKvDTmyZT7xW0yb8UesXlLu/+n1iNUmE1nMrUeocKsCCn6Wrg8cxIJxTiJhGKcREIxTiKhGCeRUIyTSCjGSSQU4yQSinESCcU4iYRinERCMU4ioRgnkVCMk0goxkkkFOMkEopxEgnFOImEYpxEQjFOIqEYJ5FQjJNIKCstn1xRFCyZOwPu9RogOCQMAPDvmZOQkZ5q3CfzehqatvDB5OiFWo1ZaoqiYMm8v45r4F/HNWsSMtIKHFdGGpo2N6/jAp4c25pFH6G2uyf8Xwk1rr8/OBBVnZyNj7v3DUXrrj21GNFk4W3dkJadi52JtwEAOh0Q0sIVXq4OsNQBOy/cwv7Ld1SbR7M4U68lYeWiObh0/jTc6zUwrr8zfa7x50vnz2DBrMkYFjlZixFNknotCSsXF3FcHxQ4rkTzOy4AuJF6Fd8vX4jki2dR293TuJ6Zlowq9g6YuOArDacznatDZbzhUwv1nKogLTvXuN7VszpqOlTGjB0XYWNlgXdfrI9rd3JxNUud6wNrFueOLbF4sc8rqOHiWuR2fV4elsydgSGj3il2H4mMx+X8X45rpHkdFwD8um0T/AIC4ejsUmg96XwCdBaW+GzqaOTmPECL9t3QY8BbsLC01GjS0vFvUB0Hk+4gKyev0Hort+dw4EoW8hUgJy8fR1Oy4fd8tWc/zr/PGiePHSly+55tm1HdyRltO/mrOVaZGY8rvpjj2m6exwUAA0ZMAAAknvy90Hp+vgGNmrdGYFgE8g16LJ89CTa2dugaFKLFmKW29sR1AIBXTftC645VrHGnQLB3cvJQp2pl1eYS+4XQTz+sQf83hmo9Rrl7Fo+rfY9gDBgxAZVtbGFr54BuwYOQEHdA67HKzEJX+M4dOh2Qr+KtPFQ7c67/ehniDz/5B2vTvgsGDYkodt+ki+dhMBjQtEVrtcYzWamO69Jfx9Vc/nEBwLa1MTh99BAAoJlvR/QJHV7kfkf3bYebRwPU9njyGVtRFFhYafpdY4mCvVzQsrYDAOCP9PvYUsx9WG7n5KGa7f8fRzUbK9z5M6/IfSuCan+Dg4ZElPiLW9DZU8fRrFUb6HS6Cp6q7Ep9XC3N47gAoE/o8GKDLCgjOQmnjuxH+MTZ0Ov1+HXbRrTu0kOFCU2z5UxmsUEWdDLtHjrWc8TJ9PuobGUBX/eq+PZYugoTPiHy5e16Wgqca9bSeoxydz01Bc6uz95x9RoUjh9WLMTcCYNhMBjQsn03+AUEaT1Wme27nAVn+0qY3rMBLC10OHAlCxdu5qj2/LwF4DOEtwA0T7wFIJGZYZxEQjFOIqEYJ5FQjJNIKMZJJBTjJBKKcRIJxTiJhGKcREIxTiKhGCeRUIyTSCjGSSQU4yQSinESCcU4iYRinERCMU4ioRgnkVCMk0goxkkkVImXxszVqzmKes6lPZuX/Nx26dm9fGT0v8zrVoml8eeJxUWu88xJJBTjJBKKcRIJxTiJhGKcREIxTiKhGCeRUIyTSCjGSSQU4yQSinESCcU4iYRinERCMU4ioRgnkVCMk0goxkkkFOMkEopxEgnFOImEYpxEQjFOIqGstHrirT9uxjdfroROp4ONrS0mT5kKr2beyM3NxUezZ+J0QgIURYF38+Z47/3psLGx0WrUUlMUBZ/Pm4G69RogaGAYAGDBrEm4kZ5q3CczIw1Nm/tg4izzuuSjoig4tGoBqtX2QLMeAwAA+sePELd+KW5dvQAFCpw9XkC7QaNgVamyxtOWXu9OXpgVGYzKlaxw+mIaImauwf2HuZrMosmZ82rSFSycPw9Ll8cgduNmjHh7JKLGRQIAYr74HAaDAd9v2oLvN23Bo9xHWLniCy3GNElachJmTxqJuIO7C61HfTAXc5atwZxla/A/E6bCzt4B4WMmazSlae5eT8bOT6fg2olfC62f2r4O+QYDgqcuQfDUJdDnPUbCjliNpjRdDUd7fDHzTYROjEGLftFISr2N6LHBms2jyZnTulIlTJ81G87OLgCApl7NcOvWLeQ9fgyfNr6o7eYGC4snrxuNmzTB5UuXtBjTJDu2xMK/zytwcnEtcrs+Lw9L583AWxHvoEYx+0h1/sBWNOzYC3bVXQqt12zgDXsnF+gsLKAD4FTHE3evJ2szZBkE+DXGsTPXcDn5JgBg+YaD+H39FIz/WJsXGk3idHOrAze3OgCevE2aP/djdPPvDutKldChYyfjfunpafhu9TeYNiNaizFNMvSvs+GpY0eK3L5n+2Y4OjmjbSd/NccqF36DRgEA0s+dKLTu1tTH+POD2zdwdu9mtH89UtXZykMdV0ek3rhrfJyWeRdVHWzhYGejyVtbTb8QysnJwcSocUhJTsb0WbMLbTt75jTCw97Aa6+/ia7dzO8XuTg/b1yDfq8P1XqMCnE7+SK2L5iExl2DUNe7ndbjlJpOp0NRdycxGPI1mEbFM+eSRZ9i/949AICu/t3Rf8BAjB0dgXr16yPmq1WFvvDZ9vNP+Ch6JqZMnYaXAoPUGtEksd8sw7HDBwAArdt3QcjgiGL3Tbp0HvkGA5o2b63WeGVy4sfVSEmIAwDU9W6HVkFhxe6bFL8fR9YtQbtBI+Hpa54vpikZd+Dr7WF87OZSFVnZD5GT+1iTeVSLc3TkOIyOHAcAePjwAQb274vgvv0QMWpMof327d2DOR/PxrIVK+HVzFut8UwWMjiixCALOnfqOLxatoFOp6vgqcpHq6CwEoP8W8qpOPweuww9ImejxvONVJisYuw+fA6fRPVDfXdnXE6+ieGvdsbWfQmazaPJZ851a77D9fR07PllF/b8ssu4vvzLr7Fg/hxAUTDzg/eN6y1b+eC9adO1GLVcZaSlwLlmLa3HKHfxG2OgQMFv331mXHPxbAK/10ZrOFXp3bzzAG/P+BZr5g1DJSsrXEm9heHTVmk2D28B+AzhLQDNE28BSGRmGCeRUIyTSCjGSSQU4yQSinESCcU4iYRinERCMU4ioRgnkVCMk0goxkkkFOMkEopxEgnFOImEYpxEQjFOIqEYJ5FQjJNIKMZJJBTjJBKKcRIJxTiJhCrxurVEpB2eOYmEYpxEQjFOIqEYJ5FQjJNIKMZJJNT/ATfse+oVq8lVAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create a random policy\n",
    "random_policy = np.ones([env.nS, env.nA]) / env.nA\n",
    "\n",
    "# run mc policy prediction for 100 episodes for random policy\n",
    "V_pi = mc_policy_eval(\n",
    "    random_policy,\n",
    "    env,\n",
    "    discount_factor=1.0,\n",
    "    episode_count=100)\n",
    "\n",
    "# Print policy\n",
    "grid_print(V_pi.reshape(env.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOcAAADnCAYAAADl9EEgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAWGklEQVR4nO3de1iUdd7H8fcAg/Rs6nohh7KDupXiocxyNzERzEpKQSWlXCqPeXbdtEw8Um27HtI8ooaahofEZBXTXQFRMbWytl2tbWs9gsjJ0ypgDsM8f1gkD9aKytw/ej6v/+a+b675/pS3M3N7XfOzuVwuFyJiHA+rBxCRK1OcIoZSnCKGUpwihlKcIoby+qmTN90/3F1zuNXbCa9YPUK16HR3gNUjVJtf/o/d6hGqjc+PVKhXThFDKU4RQylOEUMpThFDKU4RQylOEUMpThFDKU4RQylOEUMpThFDKU4RQylOEUMpThFDKU4RQylOEUMpThFDKU4RQylOEUMpThFDKU4RQylOEUMpThFD/eRXY7pL54eb8+qICGp5e3Hgm+MMjlvFuaILVo91zVwuFxsXTsX/9sa07dKr0vm1syZT+5e+hPcdacF0127r5hTWJC7Dho1aPj6MHDOOps1a4HQ6WfDWdD7e8yFOZynRMX2IjIq2etyrtillA8uXLsFms+Fz002MHTee5i1acuHCBd54PY4D+/fjcrloee+9xE6YjI+Pj1vmsvyVs369m1kUF8MzLyVwX/fXOJx9ktdGRlg91jUrOH6UxD+M4Z8fZ17x/O6UNWR9td/NU12/Y0cOEz/nTabPWcSSVe/zXP9BTHx5FAAb1yeRdewoy9Yks2j5GtatTuSfX9SMNR45fIhZM6azYHECa9dvYOCgIbz4uxEAJCyKx+l0si55I+uSN/LthW9Z8vYit81meZydHmrKp18c5eCxAgAWJ2XydHgbi6e6dvu2bqBV2BME/Sak0rkjX37Owb9/QutHulow2fWxe3vz8oQ4fOv7AdAkqDmnThbicDjI3J5GeNdueHl5UbtOXTo+1pmtW1Isnvjq2L29mfzq6/j5+QPQrHkLCgsLcVy8SOsH2zBw0BA8PDzw9PSkaVAQJ3Jy3Dab5XHeFliP7Lwz5Y+P55+hbu2bqP0L97x1uNHC+46kZbtHKh0/d7qQv66YT7dhsXh4WP7HXmW33NqAtg93AC69bZ8/axrtQsKw2+0U5OXhHxBYfq2ffwAFeXlWjVolDRrcRkiHUODSumZM+yOhYR2xe3sT3O5hGjZsBEBOznFWvrucRx/v7LbZLP/MabPZuNL+vU5nmQXTVA9naSnr5/6Bx54dSu16vlaPc11KSor5Y9wECvJymTZnIQBlrjJsNlv5NS4XeHh6WjXiNSkuLmbS+FfIzc1lwaKECue+/OIAvx85nKd7x9AhNMxtM1keZ1buadq0bFj+uIF/XU6dLaL4wkXrhqqC7UnL+PqzPQDc07otoT37VrrmxKF/cSb/BKmJ8QCcP3MKV1kZpY6LdH1hjFvnrYolC+exe2cGAMEhYXTpFsW4F4dxZ8PGvBW/lFrf3RgJCLiFwoL88p87WZiPv7+5+7bMnzubHRnbAOgQ1pEeUT0ZOWwwjX71KxKWrahww2fL5g9447U4xo2fyBNd3PtxxPZT2867YyMjv3o380lSLI/0m8XBYwW8OiKCAN86DJqSWG3P6Y6NjDYsnIr/bY2ueLd2x7rlFJ87e8Pv1lbnRkbFRUX06x1F5y4R9Bk4tMK5999bySd7d/P69NmUlBQztN9vGf3KJFo9cOPuHVTXRkZFRefp2SOSiMjuDB5a8fd9e8Y2pkwaz/z4xTRv0bJanh9+fCMjy185C06fZ9CURFZN74+3lxeHsgsZMHGF1WPJ/7F+7SrycnPIzEgnMyO9/PjMBUuIjIomJzuL/r2jcJQ6iOje84aGWZ3WrFrJiZwctqWlsi0ttfz44qXvMHPGVHC5iJs0ofx4q/tbEztxsltms/yV0wraArDm0RaAImIMxSliKMUpYijFKWIoxSliKMUpYijFKWIoxSliKMUpYijFKWIoxSliKMUpYijFKWIoxSliKMUpYijFKWIoxSliKMUpYijFKWIoxSliKMUpYijFKWKon/xqzOX7stw5i9s8EFjP6hGqRf06taweodqUOn/017TGu62e9xWP65VTxFCKU8RQilPEUIpTxFCKU8RQilPEUIpTxFCKU8RQilPEUIpTxFCKU8RQilPEUIpTxFCKU8RQilPEUIpTxFCKU8RQilPEUIpTxFCKU8RQilPEUIpTxFCKU8RQXlY+ucvlYtOiafjd3oiHnuxV6fy6WVOoXc+Xx/uMsGC6a+dyuZg3dTJ3NL6LyF7PAeB0OkmYO5Uv//4ZAK1/047nBo3CZrNZOWqVbN2cwprEZdiwUcvHh5FjxtG0WQucTicL3prOx3s+xOksJTqmD5FR0VaPe9VSt6SwduU72GyX1jX8xXE0CWqO0+lk4ZwZfLJ3F06nk169+9C1R+Xf0+piWZyFx4/y13fmknPwK/xub1Tp/J6U98j6136aPRTq/uGuQ/bRw7w9509889UB7mh8V/nxHakfkJN1lJkJ7+FylRE7sh97dqYR3OFRC6e9eseOHCZ+zpskJCbhW9+PvR/uZOLLo0jalMbG9UlkHTvKsjXJlBQXMbRfDPc0bUZQ85ZWj/1fZR09zOJ5M1m4fC2+9f34aPdOprwyitUbUtmUnET2sSMsWZlMcXERIwbGcHeTIJq6aV2Wva39NHUjrULDafrrkErnjn75OYf+8QmtH+liwWTXZ8uGtXR6ohttQzpVOF5WVsa3F0oodVzE4XBQ6nBgt9ecb2i3e3vz8oQ4fOv7AdAkqDmnThbicDjI3J5GeNdueHl5UbtOXTo+1pmtW1Isnvjq2O3ejI79YV33NP1hXbt2pNO5Szc8v1tXWKdw0v6yyW2zWfbK+f1b1UP7P61w/NzpQlLfXcDTL/+Rz7a57w/iRhk4ciwAn+/bW+F42ONd2bMzjYHR4TidTu578CHaBFf+h8lUt9zagFtubQBcets+f9Y02oWEYbfbKcjLwz8gsPxaP/8ADn7ztVWjVkngrQ0IvGxd8bOn07b9d+vKz8XvsnXV9w/g0L/dty6jbgg5S0v587w36BQzhJvr+Vo9zg21dsVi6tStx5J1qSxes5nz/znLxrXvWj1WlZWUFDN53GiOZ2fx0oQ4AMpcZRU+O7tc4OHpadWI16SkpJhXx48mJzuLMbFTACgrc2HjsnsCLhcenu5Lxm2vnDvWvcM3n+4B4O4H2tLhqT6Vrjlx+GvO5J8gLXEhAEVnT1FWVkap4yJPDhztrlGrZPWyePbt2QnAg21DeKbvkCte99GuDPoPfwm73Y7dbif0sS7s3ZlORK9n3TlulSxZOI/dOzMACA4Jo0u3KMa9OIw7Gzbmrfil1PLxASAg4BYKC/LLf+5kYT7+/gGWzHw1li2ex57M7QC0bR/KExFRTBgznDsaNubN+UsuW1cgJwsLyn/uZGEBfm5cl9vi7PBUnysGebnb7m7GiLmryx/vfH85Jef+Y/Td2mf6DvnRIC/X+O6m7N6RSsv721Ba6mDfnp3c08zsGyb9Bw+n/+DhABQXFdGvdxSdu0TQZ+DQCte16xDG5o3JBLcPpaSkmPStWxj9yiQrRr4qfV8YTt8XfljXC89G8dgTkTw3oOLfY3BIGFtSkmn7cAdKSorJSN3CqLET3Tanpf+V8v9J3yEv8vbcaYzo0wMPD0/ubd2GyOjnrR7rqq1fu4q83BwyM9LJzEgvPz5zwRIio6LJyc6if+8oHKUOIrr3pNUDbSyc9ur9ed1q8nJPsGtHOrt2/LCu6fMSiOgRTc7xbAY++xSlDgdduvfkvtbuW5f25/wZ0f6cNZP25xSpYRSniKEUp4ihFKeIoRSniKEUp4ihFKeIoRSniKEUp4ihFKeIoRSniKEUp4ihFKeIoRSniKEUp4ihFKeIoRSniKEUp4ihFKeIoRSniKEUp4ihFKeIoX7yqzFzzlx05yxu4+FRc7bdq4rCc99aPUK1+TjnlNUjVJt+be644nG9cooYSnGKGEpxihhKcYoYSnGKGEpxihhKcYoYSnGKGEpxihhKcYoYSnGKGEpxihhKcYoYSnGKGEpxihhKcYoYSnGKGEpxihhKcYoYSnGKGEpxihhKcYoYysuqJ07dksKaxHew2Wz4+PgwYvQ4mgQ1x+l0Ej97Bh/v3YXT6ST6t32I6NHLqjGrbOvmFNYkLsOGjVo+PowcM46mzVrgdDpZ8NZ0Pt7zIU5nKdExfYiMirZ63CpzuVzMmzqZOxrdRWT0cwA4nU4S5kzly398BkDrX7fjucGjsNlqzleQulwuNi+aTv3bG/GbJ3tWOp/81hRurufLo8+PcNtMlsR57OhhFs6dyeIVa/Gt78feD3cyaewo3tuYSkpyEtnHjrBsVTLFxUUMGxDD3U2CCGre0opRq+TYkcPEz3mThMSk8nVNfHkUSZvS2Lg+iaxjR1m2JpmS4iKG9ovhnqbNasS6vpd99DBvz/4T33x1gDsa3VV+fEfqB+RkHWVmwnu4XGXEjujHnh1pBIc+auG0V6/w+FFSl8/jxMGvqH97o0rnP9r0Hln/OkDQQx3cOpclb2u97d6MiY3Dt74fAE2CmnPqZCEOh4NdO9Lp3LUbnl5e1K5Tl46PhpP6l01WjFlldm9vXp5w5XVlbk8jvGs3vL5f12Od2bolxeKJq2bLn9fS6clutO3QqcLxsrIyvr1QQqnjIg6Hg1KHA7t3LYumrLq/pW3kvtBwmvy6faVzx778O4f+sY9WHbu4fS5LXjkDb21A4K0NgEtvJxbMnk5w+zDsdjv5ebn4+weWX+vnH8Chf39txZhVdsutDbjlsnXNnzWNdiGX1lWQl4d/QMV1HfymZqzrewN/NxaAz/ftrXA87PGu7NmRxsBe4TidTu578CHaBIdYMeI1+f6t6uH9+yocP3e6kLTEBfR66Q0+3/aB2+ey9IZQSUkxcbGjOZ6VxUvjpwDgKnPBZZ9VXC4XHh41675VSUkxk8eN5nh2Fi9NiAOgzFVW4TOYywUenp5WjXhDrV2xmDp167Hk/VQWv7eZ8+fOsnHtu1aPdV2cpaWkzH+DR347mJvr+Voyg9teOZcumsfuzO0ABLcP5cnIKGJHD+fOho2ZtWAJtXx8APAPDORkYUH5z50sLMDPP8BdY1bZkoXz2L0zA4DgkDC6dIti3IvDuLNhY96KX1q+roCAWygsyC//uZOF+fgbvC6A1cvi2bd7JwAPBofwTN8hV7zuo8wM+o94Cbvdjt1uJ/SxLuzdmU5Er2fdOe5Vy1z3Dv/+bA8Ad7VuS/un+lS6Jvfw15zJz2XbyoUAFJ09TVlZGaUXLxI+cLRb5nRbnP0GDaffoOEAFBcVMSAmisefjOT5ARX/wtuFhLElJZnghztQUlLMttQt/H7sRHeNWWX9Bw+n/+Af1tWvdxSdu0TQZ+DQCte16xDG5o3JBLcPpaSkmPStWxj9yiQrRr5qz/Qd8qNBXq7x3U3ZvT2Vlve3obTUwb7dO7knyNwbXe2f6nPFIC/X4O5mDJ2zqvzxrvdXUHL+7M//bm1y0mryck+QuT2dzO3p5cffnJ9AZI9ocrKz6R/zFKUOB12796RV6zZWjFll69euIi83h8yMdDIzfljXzAVLiIyKJic7i/69o3CUOojo3pNWD9SMdf03fYe+yNtzpjHi+R54eHhyb+s2RD79vNVj1XjaAvBnRFsA1kzaAlCkhlGcIoZSnCKGUpwihlKcIoZSnCKGUpwihlKcIoZSnCKGUpwihlKcIoZSnCKGUpwihlKcIoZSnCKGUpwihlKcIoZSnCKGUpwihlKcIoZSnCKGUpwihvrJr8a8UOrOUdzn1Pmf51d+ZhzM/+8X1VADBvzJ6hGqTcnf5l3xuF45RQylOEUMpThFDKU4RQylOEUMpThFDKU4RQylOEUMpThFDKU4RQylOEUMpThFDKU4RQylOEUMpThFDKU4RQylOEUMpThFDKU4RQylOEUMpThFDKU4RQzlZdUTb0rZwPKlS7DZbPjcdBNjx42neYuWXLhwgTdej+PA/v24XC5a3nsvsRMm4+PjY9WoVZK6JYU1ie9cWpePDyNGj6NJUHOcTifxs2fw8d5dOJ1Oon/bh4gevawet8pcLhcbFk7F//bGBHepPP/aWZOp/UtfwvuOtGC669f54ea8OiKCWt5eHPjmOIPjVnGu6IIls1jyynnk8CFmzZjOgsUJrF2/gYGDhvDi70YAkLAoHqfTybrkjaxL3si3F75lyduLrBizyo4dPczCuTOZNnshCYnriOn7ApPGjgIgJTmJ7GNHWLYqmYXLVrNuzbv884v91g5cRQXHj/LuH8bwz48zr3j+w5Q1HPuqZq3pcvXr3cyiuBieeSmB+7q/xuHsk7w2MsKyeSyJ0+7tzeRXX8fPzx+AZs1bUFhYiOPiRVo/2IaBg4bg4eGBp6cnTYOCOJGTY8WYVeZt92ZMbBy+9f0AaBLUnFMnC3E4HOzakU7nrt3w9PKidp26dHw0nNS/bLJ44qrZt3UD94c9QbPfhFQ6d+TLzzn490944JGuFkx2Y3R6qCmffnGUg8cKAFiclMnT4W0sm8eSOBs0uI2QDqHApbdJM6b9kdCwjti9vQlu9zANGzYCICfnOCvfXc6jj3e2YswqC7y1AW0fvvSL63K5WDB7OsHtw7Db7eTn5eLvH1h+rZ9/AIX5eVaNek3C+46kZbtHKh0/d7qQv66YT/dhsdg8au5tjNsC65Gdd6b88fH8M9StfRO1f2HNRyrLPnMCFBcXM2n8K+Tm5rJgUUKFc19+cYDfjxzO071j6BAaZtGE16akpJipr04gPy+PabPjAXCVucBmK7/G5XLhUYN/kb/nLC3l/bl/4LFnh1K7nq/V41wXm83GlXYncTrLLJjGjXHOnzubHRnbAOgQ1pEeUT0ZOWwwjX71KxKWrahww2fL5g9447U4xo2fyBNdzH6btHTRPHZnbgcguH0oT0ZGETt6OHc2bMysBUuo9d26/AMDOVlYUP5zJwsL8PMPsGDiq5eRtIyvP9sDwD2t2xLWs2+la3IO/Ysz+SfYmnjpH6HzZ07hKiuj1HGRri+Mceu81ysr9zRtWjYsf9zAvy6nzhZRfMGavXUs2cioqOg8PXtEEhHZncFDh1c4tz1jG1MmjWd+/GKat2hZLc9fXRsZFRcVMSAmisefjOT5AUMqnFu/diX7PtrDa1PfoqSkmGEDYvj92Im0an3jPtO4ayOjDQun4ndboyverd2+bjkl587e8Lu17tjIyK/ezXySFMsj/WZx8FgBr46IIMC3DoOmJFbr8/7YRkaWvK1ds2olJ3Jy2JaWyra01PLji5e+w8wZU8HlIm7ShPLjre5vTezEyVaMWiXJSavJyz1B5vZ0Mrenlx9/c34CkT2iycnOpn/MU5Q6HHTt3vOGhinXr+D0eQZNSWTV9P54e3lxKLuQARNXWDaPtgD8GdEWgDWTtgAUqWEUp4ihFKeIoRSniKEUp4ihFKeIoRSniKEUp4ihFKeIoRSniKEUp4ihFKeIoRSniKEUp4ihFKeIoRSniKEUp4ihFKeIoRSniKEUp4ihFKeIoRSniKEUp4ihfvJ7a0XEOnrlFDGU4hQxlOIUMZTiFDGU4hQxlOIUMdT/Ais0PjGaGyBPAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# run mc policy prediction for 10000 episodes for random policy\n",
    "V_pi = mc_policy_eval(\n",
    "    random_policy, env,\n",
    "    discount_factor=1.0,\n",
    "    episode_count=10000)\n",
    "\n",
    "# Print policy\n",
    "grid_print(V_pi.reshape(env.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOcAAADnCAYAAADl9EEgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAATv0lEQVR4nO3de1xUdf7H8dcgg/TY1PWHjDzSSu1iRJZR7ip5gUzTvJCaWhuVlGYW2EX7magp1dairi55QRTv5g3LlMo2xEBdtbS2Xc0tWy8ochG8rQLmMJzfH248YrFWUuZ84fd+/jfnHB7zOQ4vzszx8Zivw7IsCxExjo/dA4jIxSlOEUMpThFDKU4RQylOEUP5/tzOq+6M8dYcXjUv5RW7R6gR997gsnuEGvM/V/vZPUKN8f+JCnXlFDGU4hQxlOIUMZTiFDGU4hQxlOIUMZTiFDGU4hQxlOIUMZTiFDGU4hQxlOIUMZTiFDGU4hQxlOIUMZTiFDGU4hQxlOIUMZTiFDGU4hQxlOIUMZTiFDHUz341prf06BjCa7F9qe/ny57vjvJM/HLOFJ+ze6xfzLIs1s9JwHVtKzr0HlRl/+rpE2nw6wB6Ro+0YbpfLn1DGiuXLcLhcODv70/sqLG0Dg7B4/GQlDiVz3dsxePxMPjRIfTtX/W8TfVB2joWL5h/4byuuooxY8cRclsbzp07x5tvxLNn924sy6LN7bcTN34i/v7+XpnL9itnk8ZXkxwfxSMvp3BHv9c5mHOc10f2tXusX6zwaDbLfj+af3y+5aL7t6Wt5Mg3u7081eU7nH2QOTOmMTlxDinL1hAV/TSvjnkBgLS1qeQcPsTC5WuZs3AFa1Yu5R9f145zPHTwANOnTmH23BRWv7eOYcNH8NLzsQCkJCfh8XhYs3Y9a9au5/tz3zN/XrLXZrM9zvva38IXX2ez/3AhAHNTt/Bwz3Y2T/XL7fpkHW0jHiD4t52r7Du09yv2/20noV372DDZ5fFz+jE6Lp6AJoEAtA4O4cTxItxuN1uzMujR50Hq+frSoGEj7u3Wk/SPP7B54kvj9PNj4mtvEBh44Qu5bw25jaKiItznzxN6dzuGDR+Bj48P9erV45bgYPJyc702m+1xNg9qTE7BqYrHR4+dolGDq2jwK++8dbjSekaPpM09XatsP3OyiD8vmcWDz8Xh42P7P3u1BV3TjA4dL/zBsSyL2YlTCOsUgdPp5FhBPi5XUMWxga6mFB0rsGvUamnWrDmdu4QDF85r6uS3CI+4F6efH2H3dKRFi5YA5OYe5Z2li+l2fw+vzWb7Z06Hw8HF1u/1eMptmKZmeMrKeG/G7+n+2LM0aBxg9ziXpbS0hITXxnOsoIDJiUkAWOUWOBwVx1iWVev+AJWUlPDquFfIz89ndnJKpX17v97DiyNjePh3UXQJj/DaTLbHeST/JO3atKh43MzViBOniyk5d96+oaohM3Uh+77cDsDNoR0IHxhd5Zi8A99y6lge6csu/DKfPXUCq7ycMvd5+jw92qvzVseC5Jls25IJQFincHpFDiBuVAzXt2jF9Nnzqf/vGyOuoCCOFxVW/NzxokICXU1tmPjSzJqRSNanmwDoEnEv/QcMZORzz9DyhhtIWbik0g2fDR99yJuvxzN23AQe6O3djyOOn1t23hsLGQU2vpqdqXF0fXI6+w8X8lpsX5oGNGT4pGU19pzeWMho3ZwEXM1bXvRubdaaxZScOX3F79bW5EJGJcXFDI0awP29Inli6IhK+95b/Q67PtvO6wl/orS0hOeGRvHimAm0Db1y9w5qaiGj4uKzDOwfSd/IfjzzbOXf98xPNzHp1XHMSppLyG1tauT54acXMrL9yll48izDJy1j+ZSn8PP15UBOEUMnLLF7LPkPa1NXUJCfx5bMDLZkZlRs/+OsFCL7DyY3J4enoh6izO2mT7+BVzTMmrRy+Tvk5eayaWM6mzamV2yfu2AR06YmgGUR/+r4iu1t7wwlbsJEr8xm+5XTDloCsPbREoAiYgzFKWIoxSliKMUpYijFKWIoxSliKMUpYijFKWIoxSliKMUpYijFKWIoxSliKMUpYijFKWIoxSliKMUpYijFKWIoxSliKMUpYijFKWIoxSliKMUpYqif/WrMJbuOeHMWrwkNamz3CDWiLn99ZF12za8v/rrpyiliKMUpYijFKWIoxSliKMUpYijFKWIoxSliKMUpYijFKWIoxSliKMUpYijFKWIoxSliKMUpYijFKWIoxSliKMUpYijFKWIoxSliKMUpYijFKWIoxSliKMUpYihfO5/csizSkifjurYl7XsNqrJ/zfRJXN04gB5DYm2Y7pezLIuZCRO5rtWNRA56HACPx0PKjAT2/u1LAEJ/ew+PD38Bh8Nh56jVkr4hjZXLFuFwOPD39yd21FhaB4fg8XhISpzK5zu24vF4GPzoEPr2r/p6msrU87ItzqKj2Xy8aAa5+7/BdW3LKvu3p63iyLe7CW4f7v3hLkNO9kHmvf0HvvtmD9e1urFie1b6h+QeyWZayiosq5y4kU+yffNGwrp0s3HaS3c4+yBzZkxj7pLVBDQJZMdfNvPqmBdYtT6dtLWp5Bw+xMLlaykpKea5oVHc1DqY4JA2do/9X5l8Xra9rd2Vvp624T0J/k3nKvuy937F/r/v5M6uvW2Y7PJsWLea+x54kA6d76u0vby8nO/PlVLmPo/b7abM7cbprG/TlNXn5/RjdFw8AU0CAWgdHMKJ40W43W62ZmXQo8+D1PP1pUHDRtzbrSfpH39g88SXxuTzsu3K+cNb1YO7v6i0/czJIj5ZOpuH//ct/rqpdrzAPzZs5BgAvtq1o9L2iPv7sH3zRoYN7onH4+GOu9vTLqzqHyZTBV3TjKBrmgEX3rbPTpxCWKcInE4nxwrycbmCKo4NdDXlwD/32TVqtZh8XkbdEPKUlfH+zDfpFjWCBo0D7B7nilq9ZC4NGzVm/pp05q78iLP/Os361UvtHqvaSktLiI8bxdEjR3h53CQArHILfvTZ2bIsfHyM+tX6r0w8L69dObPWLGLfF9sBuPmuDnR5aEiVY/IO7uPksTzSl80BoPj0Cazycsrc5+k9bJS3Rq2WFQuT2LV9MwB3d+jMI9EjLnrcZ1s/5amYl3E6nTidTsK792bH5gz6DnrMm+NWy4LkmWzbkglAWKdwekUOIG5UDNe3aMX02fOp7+8PgCsoiONFhRU/d7yokEBXUxsmvjS15by8FmeXh4ZcNMgfa37TrYycsaLi8eZ3F1Ny5l9G3619JHrETwb5Y61uuoVtWem0ubMdZWVudm3fzM23mn3D5MnhMTw5PAaAkuJihkYN4P5ekTwxtPL53tM5gg1pawnr2IXS0hI2pW/gxTET7Bj5ktSW87L1v1L+P4ke8RLzZkwmdkh/fHzqcXtoOyIHP2H3WJdsbeoKCvLz2JKZwZbMjIrtf5yVQmT/weTm5PBU1EOUud306TeQtqHtbJz20pl8Xlqfsw7R+py1k9bnFKllFKeIoRSniKEUp4ihFKeIoRSniKEUp4ihFKeIoRSniKEUp4ihFKeIoRSniKEUp4ihFKeIoRSniKEUp4ihFKeIoRSniKEUp4ihFKeIoRSniKEUp4ihfvarMY+eOu/NWeQynSyuu6/XV/mn7B6hxkTd1fyi23XlFDGU4hQxlOIUMZTiFDGU4hQxlOIUMZTiFDGU4hQxlOIUMZTiFDGU4hQxlOIUMZTiFDGU4hQxlOIUMZTiFDGU4hQxlOIUMZTiFDGU4hQxlOIUMZTiFDGUr11PnL4hjVXLFuFwOKjv70/sqLG0Dg7B4/GQlDiVnTu24vF4GPToEPr2H2TXmNVWV8/rB5ZlMTNhIte1vJHIwY8D4PF4SHk7gb1//xKA0N/cw+PPvIDD4bBz1GqxLIv1cxJwXduKDr2rvi6rp0+kwa8D6Bk90msz2RLn4eyDJM+YRvKS1QQ0CWTHXzYzccwLrFyfzgdrU8k5fIgFy9dSUlJMzNAobmodTHBIGztGrZa6el4/yMk+yLzEP/DdN3u4ruWNFduz0j8k90g201JWYVnlxMU+yfasjYSFd7Nx2ktXeDSbjxe+zdH93+C6tlWV/dvSVnLkm93c2j7cq3PZ8rbWz+nH6Lh4ApoEAtA6OIQTx4twu91szcqgR58HqefrS4OGjYjo1pONH39gx5jVVlfP6wcb3l/Nfb0epEOX+yptLy8v5/tzpZS5z+N2uylzu3H61bdpyurb9ck62kY8QPBvO1fZd2jvV+z/205Cu/bx+ly2XDmDrmlG0DXNgAtvJ5ISpxDWKQKn08mxgnxcrqCKYwNdTTnwz312jFltdfW8fjDs+TEAfLVrR6XtEff3YXvWRoYN6onH4+GOu9vTLqzqL7qpfniremD3rkrbz5ws4s9LZvG7MX/gywzv/yG19YZQaWkJ8XGjOHrkCKPHTQLAKrcqfVaxLAsfn9p136quntdPWb1kLg0bNWb+u+nMXfURZ8+cZv3qpXaPdVk8ZWW8N+P3dH/sWRo0DrBlBq9dORcmz2TblkwAwjqF80DkAMaNiuH6Fq2YNns+9f39AXAFBVFUVFjxc8eLCgl0NfXWmNVWV88LYMXCJHZt2wzA3WGdeSR6xEWP+2zLpzwV+zJOpxOn00l4997s2JxB30GPeXPcS5aZupB9X24H4ObQDoQPjK5yTN6Bbzl1LI/0ZUkAnD11Aqu8nDL3efo8Pdorc3otzujhMUQPjwGgpLiYYVED6N4rkieGVn7BwzpHsCFtLWEdu1BaWsKn6Rt4ccwEb41ZbXX1vAAeiR7xk0H+WKubbmFbZjpt7mxHWZmbXds2c3OwuTe6wgdGXzTIH2t+cwjPz1xZ8ThrzWJKzpyu+3dr309dQUF+HlszM9iamVGxfeqsFCL7DyY3J4ehUQ9R5nbTu99A7ghtZ8eY1VZXz+u/iX72Jea9PZnYJ/rj41OP20PbEfnwE3aPVetpCcA6REsA1k5aAlCkllGcIoZSnCKGUpwihlKcIoZSnCKGUpwihlKcIoZSnCKGUpwihlKcIoZSnCKGUpwihlKcIoZSnCKGUpwihlKcIoZSnCKGUpwihlKcIoZSnCKGUpwihvrZr8YsdXtzFO85UUe/QjLjnwV2j1Bjhg9LsHuEGlP615kX3a4rp4ihFKeIoRSniKEUp4ihFKeIoRSniKEUp4ihFKeIoRSniKEUp4ihFKeIoRSniKEUp4ihFKeIoRSniKEUp4ihFKeIoRSniKEUp4ihFKeIoRSniKEUp4ihfO164g/T1rF44XxwOPD3v4oxY8cRclsbzp07x1tvxLNnz24sy6JNm9sZO34i/v7+do1aLekb0li1bBEOh4P6/v7EjhpL6+AQPB4PSYlT2bljKx6Ph0GPDqFv/0F2j1ttlmWRljwZ17Utad+r6vxrpk/i6sYB9BgSa8N0l69HxxBei+1LfT9f9nx3lGfil3Om+Jwts9hy5Tx08ADT/ziFWckprH53HcOGj2DUCxdezJS5SXg8HlLfW0/qe+v5/vvvWZCSbMeY1XY4+yDJM6aRkDiHecvWEBX9NBPHvADAB2tTyTl8iAXL15K0cAXvrlzKP77ebe/A1VR0NJt33nyZbz7fctH929NWceTb2nVOP9ak8dUkx0fxyMsp3NHvdQ7mHOf1kX1tm8eWOJ1+frwa/waBgS4AQkJuo6ioCLf7PKF3tWPY8BH4+PhQr149WgcHk5uba8eY1ebn9GN0XDwBTQIBaB0cwonjRbjdbrZmZdCjz4PU8/WlQcNGRHTrycaPP7B54urZlb6etuE9Cf5N5yr7svd+xf6/7+TOrr1tmOzKuK/9LXzxdTb7DxcCMDd1Cw/3bGfbPLbE2axZczp3CQcuvE2aOvktwiPuxen0I+yejlzfoiUAublHWb50Md2797BjzGoLuqYZ7Tte+MW1LIukxCmEdYrA6XRyrCAflyuo4thAV1MKj9Wub2jvMSSW2+7pWmX7mZNFfLJ0NpHPjsXHp/bexmge1JicglMVj48eO0WjBlfR4Ff2fKSy7TMnQGlJCRPGv0JBfj6z5qRU2rf36z289HwMgx+JonN4hE0T/jKlpSUkvDaewoICEhKTALDKLRwOR8UxlmXV6l/kH3jKynh/5pt0ixpBg8YBdo9zWRwOBxdbncTjKbdhGi/GOXtmIpmfbgIgPOJe+g0YyPPPPUPLVjcwb8GSSjd8Pv7oQ958I55Xxk3ggV59vDXiL7IweSbbtmQCENYpnAciBzBuVAzXt2jFtNnzqf/v83IFBVFUVFjxc8eLCgl0NbVh4kuXtWYR+77YDsDNd3Wgy0NDqhyTd3AfJ4/lkb5sDgDFp09glZdT5j5P72GjvDnuZTuSf5J2bVpUPG7masSJ08WUnLNnbR1bFjIqLj7LoAGR9Onbj2eejam0LytzE5NeHcfM2XMJua1NjTx/TS1kVFJczLCoAXTvFckTQ0dU2vfe6nfY9dl2Xk/4E6WlJcQMjeLFMRO4I/TKfabx1kJGaXMmE3hti4verd387mJKzvzrit+t9cZCRoGNr2Znahxdn5zO/sOFvBbbl6YBDRk+aVmNPu9PLWRky9valcvfIS83l00Z6WzKSK/YPnf+IqZNTQDLIn7i+Irtbe8MJW78RDtGrZb3U1dQkJ/H1swMtmZmVGyfOiuFyP6Dyc3JYWjUQ5S53fTuN/CKhimXr/DkWYZPWsbyKU/h5+vLgZwihk5YYts8WgKwDtESgLWTlgAUqWUUp4ihFKeIoRSniKEUp4ihFKeIoRSniKEUp4ihFKeIoRSniKEUp4ihFKeIoRSniKEUp4ihFKeIoRSniKEUp4ihFKeIoRSniKEUp4ihFKeIoRSniKEUp4ihfvZ7a0XEPrpyihhKcYoYSnGKGEpxihhKcYoYSnGKGOr/ABKmPCbNje79AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# run mc policy prediction for 100000 episodes for random policy\n",
    "V_pi = mc_policy_eval(\n",
    "    random_policy,\n",
    "    env,\n",
    "    discount_factor=1.0,\n",
    "    episode_count=100000)\n",
    "\n",
    "# Print policy\n",
    "grid_print(V_pi.reshape(env.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion\n",
    "\n",
    "We see that state values do not converge very well for 100 episode simulation. However, with 10,000 episode simulation the values have converged very well and match those we saw in `listing3_2` where policy evaluation using DP was carried out. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
